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tc the residence hall counselor’s 

studied More soeciftcally, a valid and reliable rating scale for use by .tudenU in 
evabating their resident hall counselor was developed. In developing 
for this purpose, some of the weaknesses of the more common and popcten fype 
rltino scates such as rater leniency, rater bias, and halo effect were avoided. Jhe 
forc^-choice technique was considered th e best technique in overcoming such 
weaknesses. (Data an%sis showed that an instrument composed of items significantly 
different in their ability to discriminate between effective and p,. 

counselors, yet prossessing comparable degrees of favorablenes., c°u° 
constructed from^student suppliecl descriptive behavior statements describing the 
plrf^mance and activities o" residence ^hall counselors.) This ins rument had a 

satisfactory split-half reliability of .74. There was no significant difference in the 
reliability for men and women. A reliable and valid instrument (forced choice mating 
scale), Vrough which student opinion of the residence hall counselor can be 
successfully measured, has been developed. (Author /PS) 
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Introduction 

Burgeoning enrollments have made it necessary for 
colleges and universities to house an increasing number 
of students in college-owned and operated facilities# 

This sudden concentration of students has been a potent 
factor in the changing character of the campus scene# A 
cursor^' review of the literature relevant to housing 
reveals a consensus that residence halls can make a 
significant contribution to the education of students 
beyond the mere provision of food and shelter# In fact^ 
residence halls ^ in their provision of communal living > 
are now highly committed to the furtherance of higher 
learning,^ and increased efforts are being made across 
the country to align the living unit with the classroom# 

A key person in the implementation of an effective 
housing program is the resident counselor# These are 
graduate or advanced undergraduate students assigned by 

Williamson, E. G., and vase, Max, “Symposium: 
Residence Halls in Higher Education and Residence Halls in 
Higher Learning," Personnel and Guidance Journal 3^:392- 
l^01, February, 195^* 
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the school in recognition of the fact that there is a 
responsibility not only for class and laboratory instmc- 
4.4«v» f nnrf . hii+. for thoss oorti-ons of the 

lives of the student not otherwise touched upon by the 
instructional staff but which are nevertheless a vital link 
to progress in the total academic setting. In other words, 
the residence hall counselor is particularly responsible 
for the social-educational aspects of university living in 
a setting where the student spends as much as 6$ per cent 
of his time. 

To place the residence hall counselor in the circle 
of “academic responsibility,”^ carries with it certain 
expectations from which the practitioner cannot escape. 

This ^jnmediately suggests that some means of evaluation ®f 
residence hall programs is needed. The evaluation of total 
program is important, since the success of individual per- 
formers within a program is often inferred from the total 
success of a program. However, our concern here is more 
specifically with direct evaluation of the individual 
counselor. 

Presently such evaluation as is done is limited 
primarily to observations by staff and administrative 

^Kilbourn, D. ¥., “The Status and Roles of Head 
Residents,” Personnel and Guidance Joiurnal 39s 203-206, 
November, I960. 
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officers responsible for residence hall programs,^ More 
specifically these supervisor evaluations are frequently 
obtained by means of a «free written” supervisor rating. 

This results in an over-all rating of the counselor which 
often does not reflect individual or specific areas of 
strength and/or weakness. There have been some attempts 
to rate individual counselors in terms of component com- 
petencies by means of the composite type scale. This type 
of scale usually splits the counseling process into subjec- 
tive and philosophically based units and asks the rater to 

evaluate the rates on each component unit. 

This composite type rating is, however, usually done 

by a supervisor and lacks some of the reliability and 
validity desired of performance ratings. Student evalua- 
tion, on the other hand, has been virtually unobtained 
except that which reaches the supervisor by word of mouth. 
Typically, this type of "evaluation” represents the student 
"gripe" and may be rather localized in terms of scope, thus 
not reflecting even an Jiccurate measure of the rating by 
the individual registering the "gripe" to say nothing about 
that of other counselees with whom the counselor has con- 
tact. Nevertheless, these "gripes" may find their way into 
the supervisor’s rating of the counselor, biasing an 
over-all rating in one way or another. Thia is likely to 

3sifferd, C. S., "Evaluating a Residence Hall Coun- 
seling Program," School and Society 69sii>2-U54, Jvne 25, 

19li9. 
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be the case, since the contact which the supervisor has 
with the counselor is for the most part more limited and 
presumably makes the supervisor less able to render a 
total evaluation than might be the case for the counselee 
who has constant daily contact with the counselor in rela- 
tionships that cover all aspects of his job. 

Further, the perspective with which the supervisor 
views the counselor is likely to be quite different from 
that of the student counselee. Thus, considering that the 
student is the one being served, it seems only appropriate 
that he be given an opportunity to render judgment as to 
the effectiveness with which the counselor carries out his 
job. 

This study was conceived specifically to get at 
this little sought after area of opinion with reference to 
the residence hall counselor’s performance, since it 
appears so vital to the success of any residence hall 
counseling program. As Claude W. Grant has indicated: 

One observation which seems safe to make 
about any di.scipline whose major concern is working 
with people is that the scope of operation of a 
member of the discipline is probably as dependent 
upon what people think he can do as upon what he 
has been trained to do«^ 



^Qrant, C. W., '*How Students Perceive the Counselor’s 
Role,” Personnel and Guidance Journal 32:36?, March 195U* 
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Student evaluation coiild, of course, be sought by 
means of the ’’over-all" or "free Triritten“ technique. How- 
ever, our general knowledge of the studmt*s willingness to 
devote the time required for this kind of evaluation would 
lead us to seek other means. Further, since the technique 
does not focus on specific areas of behavior, its utility 
as a measure against which to guide improvement is ques- 
tionable. 

In developing an instrument for use by students in 
evaluating the performance of residence hall counselors, 
the investigator attempted to avoid some of the weaknesses 
of the more common and popular type rating scales such as; 

1, The common occurrence of "over-rating" by raters < 

2. Ratings by some raters which are not comparable 

to those of others. 

3* Over-all generalized ratings which fail to 
bring out real differences among individuals being rated— 

halo effect* 

ii. Ratings which are not consistent and reliable 
or at least in which inadequate or a complete lack of 
reporting of reliability is evidenced. 

. Necessity for elaborate training of raters for 

any reliable ratings to be obtained. 

It was primarily to overcome some of these types #f 

^Richardson, M. W., "Forced-Choice Performance 
Reports," in Rating Employee and Supervisory Perfonn^* 
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problems that the forced-choice technique was developed by 
the Personnel Research Section of the Adjutant General *s 
Office in 19h$* Research conducted since its development 
indicates that this technique comes closer than any method 
yet devised in overcoming the weaknesses of more conven- 
tional rating techniques* Thus^ construction of such a 
scale is pursued here in hopes of the evolution of an 
empirical tool through which more objective assessment of 
the relationship between counselor and counselee can be 
made for the ultimate improvement of residence hall 
counseling— a vital adjunct of higher learning* 

Purpose of the Study 

The purpose of this study was to construct a forced- 
choice rating scale through which student evaluation of 
the performance of residence hall counselors co\ild be 
measured* 



Delimitations 

The study was limited to the development of a 
forced-choice rating scale which could be utilised to meas- 
ure the opinion of college students with reference to the 
performance of residence hall cotmselors* Only undergrad- 
uate students living in the halls ef i^sidence at Indiana 
University were used for purposes of this sttidy. 
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Definition of Terms 



Definition of the following terms should facilitate 
understanding of the procedures used in this study. 

Forced-Choice Rating Scale . A forced-choice rating 
scale is one in which the rater is required to make a 
series of choices, from groups or blocks of descriptive 
behavior items, of those items which are most and/or least 
descriptive of, characteristic of, or applicable to the 
person being rated. The point here is to force the rater 
to choose from among descriptive phrases which appear of 
equal value but which actually differ at a statistically 
significant level in their power to discriminate between 
good and poor counselors. Thus, the rater* s ability to 
control the final result of his rating is reduced. 

Descriptive Behavior Statements . The descriptive 
behavior statements are ihort phrases or sentences which 
describe qualities of residence hall counselors. These 
statements were supplied by college students. 

Preference Index . The preference index is a measure 
of the favorability of the descriptive behavior item. The 
measure of favorableness was determined by averaging scale 
values for descriptive behavior items after the behavior 
statements had been judged as to degree of favorableness 
(on a five point scale ranging from "very favorable" to 
"very unfavorable") to counselors as a whole. 



Discrimination Index . The discrimination index is 
a measure of how well the descriptive behavior item dis- 
criminates between effective (good) and ineffective (poor) 
counselors. The discrimination index was computed as the 
difference between the average desoriptiveness of an item 
when used to describe the effective counselor, and when 
usfd to describe the ineffective counselor. The descrip- 
tiveness ratings were derived from two separate ratings 
baaed on a five point scale of descriptivsnesso 

■Triad . A triad refers to the arrangement of the 
descriptive behavior statements in groups of three. All 
three have a similar preference index but vary in terms 
of discrimination itidex. A sample triad follows; 

1-A Speaks Well 
1-B Is Very Diligent 
1-C Sincere 

Significance of the Study 

The instrument should have general utility as a 
measure against which the counselor can judge his level 
of performance as far as his constituents are concerned 
and his need for growth and ii^irovement. Further, the 
supervisor should be able to utilize the results from such 
a measure in counseling counselors. These two are 
particularly important when it is recognized that many 
people involved in residence hall counseling are actually 
interns in college student personnel work or eventually 
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end up as practicing student personnel workers • Consider- 
ing that they will be working with students for some years 
to com©, It seems quite impoirtant that they have some 
measure of student reaction to their ability against which 
to gauge performance. Likewise, it seems equally important 
for residence hall administrators to he aware of the 
standards used by students in evaluating the performance 
of counselors# 

The instrument could also add to the meaning of 
ratings by supervisors# However, it is not suggested that 
the instrument developed here be used as a replacement for 
the latter. On the other hand, this research could easily 
lead to further research involving the development of 
similar instruments for use by supervisors or a single 
instrument for use both by students and supervisors# 
Another line of research would be development of a similar 
instrument to use in the recruitment of prospects ''^o coun«® 
selors. A more extended development from such research 
would be efforts toward construction of an instrument of 
general utility in the field of college student personnel# 

Summary of Procedures and Findings 

The following procedures were used to develop the 
forced "Choice rating seals for student evaluation of 
residence hall counselors# 
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The investigator first, through careful examination 
of residence hall counselor 1 ob descriptions, isolated 
categories of job activities. These categories isolated by 
the investigator were subjected to experts in the field o:f 
student personnel administration who judged their appro- 
priateness. On the basis of percentage of agreement among 
judges, the number of job activity categories was reduced 
to 10. This was an important initial step, since these 
categories could then be used to limit the universe of 
content of the rating scale being developed, thereby help- 
ing to insure rating scale validity. 

After establishment of job activity categories, 50O 
undergraduate students, all with two or more years of 
living experience in residence halls, were solicited, by 
means of a questionnaire, for both favorable and unfavor- 
able descriptive behavior statements concerning the 
behavior and activities of residence hall counselors. This 
inquiry yielded an unedited pool of 5 ,l 6 ij descriptive 
behavior statements. Editing and elimination of duplicate 
Hems reduced the total item pool to iil 6 statements. 

The remaining I 4 I 6 descriptive behavior statements 
were submitted to 12 super^^isors of residence hall coun- 
selors and members of a residence hall counseling staff-. 
These 12 experts judged the relatedness of the kl6 state- 
ments to the 10 job activity categories previously estab- 
lished. On the basis of 7!> per cent agreement among the 
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kludges , the number of descriptive behavior statements was 
further reduced to 176* 

Next, 100 student judges, all with two or more years 
of living experience in residence halls, rated each of the 
remaining 176 items on a one-to*-five scale of favorability* 
This was done to establish the preference index of each 
item. This step was vital, since the final rating scale 
was to be composed of items with an equal or comparable 
favorability rating when applied to counselors in general. 

Another 100 student judges, again with two or more 
years of living experience in residence halls, evaluated 
the 176 items in terms of their applicability to the ”best" 
(most effective) and ’’worst" (most inef ’'ective) counselor 
they had known. This rating was done on the basis of a 
five point scale. The mean rating for the "best" rating 
and the mean rating for the "worst" rating was subjected to 
a statistical test of the significance of differences 
between means in order to determine the discrimination 
index of each item. Those items significantly discrimi- 
nating between effective and ineffective counselors at ths 
one per cent level of confidence were retained for 
inclusion in the final form of the instrument. This 
process, coupled with the preference index requirement of 
comparable favorability ratings, resulted in the retention 
of 96 items for inclusion in the final form of the forced- 
choice rating scale. 
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The remaining 96 items -were arranged into 32 triads 
or blocks of three statements each for the final rating 
instrument. Each triad contained items with equal or com- 
parable preference ratings 5 however, two of the three 
statements possessed a high discrimination index while 
the third statement in the triad had a low discrimination 
index. 

The completed instrument was then printed and 
administered in a residence center not previously sampled 
in the construction phases of the study. All 23 counselors 
in this center distributed a total of 1,112 rating scales 
and received a return of 619 completed rating forms. A 
split-half reliability on the odd versus even items using 
a Spearman-Brown formula resulted in a reliability 
coefficient of .7U» Separate reliability coefficients for 
men and women were not significantly differ^^nt from the 
total test reliability* 

A similar administration was conducted by 18 coun- 
selors in another residence center among 816 students. 

From a return of li39 completed rating forms, the split- 
half reliability coefficient was .?0. 

The return of 1*39 forms was combined with the 6l9 
completed forms from the test residence center netting a 
total of l,lii7 individual ratings. On the basis of the 
results obtained from these ratings, m.rms were established 
for the forced-choice rating scale. The mean score for the 
l,lh7 ratings was 30.86 with a standard deviation of 3*32. 
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Conclusions 



This study was conceived to explore the area of 
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selor’s performance* More specifically, the purpose of the 
study was the development of a valid and reliable rating 
scale for use by students in evaluating their residence 
hall counselor. 

In developing an instrument for this purpose, the 



investigator sought to avoid some of the weaknesses of the 
more common and popular type rating scales such as rater 
leniency, rater bias, and halo effect. The forced-choice 



technitjue was selected as coming closer than any other 



technicjue in overcoming such weaknesses. 

Analysis of the data presented revealed that, from 
student supplied descriptive behavior statements describing 
residence hall counselor performance and activities, an 
instirument composed of items significantly different in 
their ability to discriminate between effective and 
ineffective residence hall counselors yet possessing com- 
parable degrees of favorableness could be constructed. It 
was further demonstrated that such an instrument, once 
constructed, had a satisfactory split-half reliability of 
,7ii. At the same time, there was no significant difference 

in the reliability for men and women. 

Thus, the investigator must conclude that a reliable 
and valid instrument (forced-choice rating scale), through 
which student opinion of the r^'s-dence hall counselor can 
be successfully measured, has been developed. 



